Systems and methods for processing of dental images

ABSTRACT

A computer system implements a neural network to process raw dental images to detect and number teeth and to diagnose pathological, non-pathological, and post-treatment conditions. Detected teeth, corresponding numbers, and any corresponding detected conditions are correlated to the dental image and presented in a graphical user interface comprising the image and a standard, symbolic dental chart associating the tooth number, detected conditions, and regions of the image to teeth represented in the symbolic chart.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/454,902 filed Jun. 27, 2019, which claims priority to U.S.Provisional Application No. 62/690,844 filed on Jun. 27, 2018, theentireties of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to image processing and, in particular,to processing of dental imaging and conversion of the raw image data anddiagnostic findings into the symbolic representations corresponding tothe international standards in dentistry.

TECHNICAL BACKGROUND

Analysis of dental imaging, such as radiographs, and recording theresults of such analysis in the form of dental charts is an importantelement of daily clinical practice. Radiographs such as panoramicradiographs, bitewings, Full-Mouth Series (FMX), and Cone-Beam ComputerTomographs (CBCT), provide images of more than one of a patient's teeth.When a dental image includes more than one tooth, one of the tasks of ahuman expert analyzing the image is to perform teeth detection andnumbering. This is preferably done according to a recognized notation,such as the FDI notation published by the International Organization forStandardization (e.g., ISO 3950:2016). Accurate analysis of dentalimaging is a precursor to the detection of pathologies and to thegeneral management of the majority of dental practices. However, theroutine nature of dental charting diverts significant time and attentionin dental practice.

Computer-aided diagnosis (CAD) has developed significantly due to thegrowing accessibility of digital medical data, rising computationalpower and progress in artificial intelligence. CAD systems assistingphysicians and radiologists in decision-making have been applied tovarious medical problems, such as breast and colon cancer detection,classification of lung diseases, and localization of brain lesions.

Despite existing approaches, barriers remain to the reliable automationof dental imaging analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

In drawings which illustrate by way of example only embodiments of thepresent application,

FIG. 1 is a schematic illustration of a workflow and select systemcomponents of a system for processing dental imaging.

FIG. 2 is a set of images illustrating successful detection of teeth bythe system of FIG. 1.

FIG. 3 is a set of images illustrating successful classification ofteeth by the system of FIG. 1.

FIG. 4 is a schematic of a possible networked environment for the systemof FIG. 1.

FIG. 5 is an illustration of an example graphical user interface on aclient system displaying a panoramic radiograph based on the detectionand classification executed by the system of FIG. 1.

FIG. 6 is an illustration of a graphical user interface on a clientsystem displaying a bitewing radiograph based on the detection andclassification executed by the system of FIG. 1.

FIG. 7 is an illustration of a further example graphical user interfacedisplaying an X-ray image based on output by the system of FIG. 1.

FIG. 8 is an illustration of an example report generated at the clientsystem based on the output by the system of FIG. 1.

FIG. 9 is an illustration of an example portion of a graphical userinterface or report generated using the output of the system of FIG. 1.

DETAILED DESCRIPTION

The examples and embodiments described in this disclosure providesystems, methods, and data processing device-readable media forprocessing dental imaging. In particular, the example system and methodsdescribed herein apply deep learning techniques to the processing ofdental images to provide a platform for computer-aided diagnosis andcharting, and in particular to detect and number teeth. Deep learning isa class of learnable artificial intelligence (AI) algorithms that allowsa computer program to automatically extract and learn important featuresof input data for further interpretation of previously unseen samples.Deep learning techniques differ from conventional image processingtechniques in that deep learning techniques can learn from a raw datainput, for example pixels of images, with no handcrafted featureengineering required. A detailed overview of deep learning techniques isavailable in LeCun Y, Bengio Y, Hinton G, “Deep Learning”, Nature, 2015;521: p. 436-444.

An example workflow for detecting and numbering teeth using deeplearning is illustrated schematically in FIG. 1. In this example, asystem implementing deep learning techniques includes a detection moduleand a classification or numbering module, both applying deep learningtechniques to detect and classify teeth, respectively, in a sourcedental image. The detection module receives the source dental image andgenerates data identifying image regions determined to correspond toindividual teeth. The identifying data can take the form of boundingboxes defining a boundary for each tooth captured in the dental image.Output from the detection module is then provided to the classificationmodule to assign a number to each detected tooth. Numbering may beassigned in accordance with the FDI notation, the Universal NumberingSystem, the Palmer notation method, or any other suitable dentalnotation. In the accompanying figures, either FDI notation or theUniversal Numbering System is employed.

In one embodiment, convolutional neural networks (CNNs) are utilized forboth detection and numbering of teeth. CNNs are a standard class ofarchitectures for deep feedforward neural networks, and they areapplicable for image recognition tasks (see e.g., LeCun Y, Bottou L,Bengio Y, Haffner P. Gradient-based learning applied to documentrecognition. Proc IEEE. 1998; 86(11): p. 2278-2323). CNN architecturesexploit specific characteristics of an image data input, such as spatialrelationships between objects, to effectively represent and learnhierarchical features using multiple levels of abstraction. Thoseskilled in the art will recognize, however, that appropriate neuralnetwork models based on architectures other than CNNs may be employed.

In the example of FIG. 1, the source image 100 is provided in a digitalformat as input to the detection module 200. If the source image isinitially produced on film, the image is initially digitized. In theexamples discussed herein, source images are panoramic view (PV)radiographs that depict the upper and lower jaws in one single image.However, those skilled in the art will appreciate that these examplesneed not be limited to PV radiographs. Other types of dental images,such as medical computed tomography scans and bitewing radiographs,intraoral three-dimensional scans, still images and videos may beemployed as input. PV and bitewing intraoral radiographs in particularare convenient because they permit screening of a broad anatomicalregion while requiring relatively low radiation doses, and are inrelatively common usage.

The detection module detects teeth in the original image. Teethdetection may comprise implementation of the Faster R-CNN modeldisclosed in Ren S, He K, Girshick R, Sun J. Faster R-CNN: TowardsReal-Time Object Detection with Region Proposal Networks. IEEE TransPattern Anal Mach Intell. 2017; 39: p. 1137-1149. Faster R-CNN is asingle unified network consisting of two modules: the regional proposalnetwork (RPN) and object detector. The RPN module proposes regions whereobjects of interest might be located. The object detection module usesthese proposals for further object localization and classification. Boththe RPN and object detector modules share the convolution layers 210 ofthe base CNN that provides a compact representation of the source image,known as a feature map 220. The features are learned during a trainingphase, unlike in classical computer vision techniques in which featuresare engineered by hand.

To generate regional proposals, the RPN module slides a window over thefeature map 220, and, at each window location, produces potentialbounding boxes named “anchors”. For each anchor, the RPN moduleestimates the probability that the anchor contains an object or abackground (e.g., employing a softmax function), and tightens thebounding box with a specialized bounding box regressor to identifyregion proposals. The top N-ranked region proposals (indicatedschematically in 230) then serve as input for the object detectionmodule. One head 240 of the object detection module carries out a binary(two-class) detection task on each region of interest thus identified,refining the determination whether the region of interest is a tooth ora background. Another head 250 of the object detection module generatesthe final bounding box coordinates, represented schematically on theoriginal input image as outlines in image 300.

A VGG-16 Net (Simonyan K, Zisserman A. Very Deep Convolutional Networksfor Large-Scale Image Recognition. In International Conference onLearning Representations (ICLR); 2015 May) can be used as a base CNN forboth RPN and object detection. The hyperparameters that define theanchor properties are preferably tuned to reflect the potentialboundaries of teeth. These hyperparameters include base anchor size,anchor scales, and anchor ratios.

Preferably, to minimize false positives during teeth detection, theIntersection-over-Union (IoU) threshold for a non-maximum suppressionalgorithm (NMS) is used in the system and prediction score threshold isalso tuned.

The output 300 of the detection module is provided to the classificationmodule 500. The classification module 500 is trained to predict thenumber of a tooth according to a notation. In the example of FIG. 1, FDInotation is used. The classification module 500 is modelled according toa CNN such as the VGG-16 convolutional architecture 510. To classify theteeth by numbers, the classification module initially crops the imagebased on the predicted bounding boxes to produce cropped images 400,which are provided as input to convolutional layers 510. Theclassification module 500 then performs a 32-class detection task, e.g.using a softmax function 520, producing probabilities that each croppedimage represents each of the 32 permanent teeth. This results in a setof confidence scores over 32 classes for each bounding box identified bythe detection module 200, which can be used to predict a tooth numberaccording to the desired (standard) notation. It should be noted that inthese examples, an adult dental chart is used, although a primary dentalchart and appropriate notation may be employed for younger patients.

However, in a post-processing stage, the classification module 500 maythen apply heuristics or other constraints to the sets of confidencescores to improve prediction results. For example, a heuristic maycomprise the assumption that each tooth can occur at most once in theimage in a specific order, to ensure arrangement consistency among thedetected teeth. In the case of bitewing images and intraoral scans, theinput data to the classification module may also include informationabout the position of the sensor (in the case of an image or scangenerated digitally) or the film, which imposes constraints on the teeththat are likely to appear in a given image.

In one embodiment, this post-processing comprises the following steps:

1. Sort predicted teeth bounding boxes by coordinates within each jaw.

2. Count the number of missed teeth based on the known maximum teethcount.

3. Iterate over all possible valid combinations of teeth and calculatethe total confidence score.

4. Choose the combination with the highest total confidence score.

The system produces as output (6) the coordinates of the bounding boxesfor the teeth detected in the source dental image, and correspondingteeth numbers for all detected teeth in the image.

EXAMPLE

A system implementing the foregoing methodology was used to process adata set of 1574 anonymized PV radiographs of adults randomly chosenfrom the X-ray images archive provided by the Reutov StomatologicalClinic in Russia from January 2016 to March 2017. No additionalinformation such as gender, age, or time of image taking was used. AllPV images were captured with the Sirona Orthophos XG-3 X-ray unit(Sirona Dental Systems, GmbH, Bensheim, Germany). Five radiology expertsof varying experience provided ground truth annotations for the images.To collect these annotations, the experts were presented withhigh-resolution PV images and asked to draw bounding boxes around allteeth and, at the same time, to provide a class label for each box withthe tooth number (according to the FDI system).

The images were randomly distributed into a training group of 1352images and a testing group of 222 images. The training group was used totrain the teeth detection and classification models, and the testinggroup was used for evaluation of the performance of the approach.

Training

During training for teeth detection, model weights pretrained on theImageNet dataset were used for the basic CNN (Deng J, Dong W, Socher R,Li L J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical imagedatabase. In The Conference on Computer Vision and Pattern Recognition(CVPR); 2009. p. 248-255). All layers of the CNN were fine-tuned sincethe dataset was sufficiently large and different from ImageNet. Theinitial learning rate was chosen as 0.001 with further exponentialdecay. The model was trained only to detect teeth with natural roots,excluding dental implants and fixed bridges.

The detection module 200 was implemented using a customized version ofthe Faster R-CNN python (Hosang J. Faster RCNN TF. 2016. Available fromgithub.com/smallcorgi/Faster-RCNN_TF) implementation with the TensorFlowbackend Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, et al.TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015.Available from tensorflow.org). The hyperparameters that define theanchor properties were tuned to reflect the potential boundaries ofteeth. To minimize the false positives rate of teeth detection, theIntersection-over-Union (IoU) threshold for non-maximum suppressionalgorithm (NMS) used in the system and prediction score threshold werealso tuned.

The classification module 500 was written using the Keras library(Chollet F. Keras. 2015. Available from github.com/fchollet/keras) withthe TensorFlow backend. As with teeth detection, for teethclassification the model weights pretrained on the ImageNet dataset wereused to initialize the CNN in the classification module. For training,cropped images were produced based on the ground truth annotations offull panoramic X-rays, and the cropping method was tuned to includeneighbouring structures, which improved the prediction quality of theCNN because of additional context. The images were also augmented toincrease the variety of the available dataset. A batch size of 64 wasused to train the CNN.

Performance Analysis

The testing group of 222 images was used to evaluate the performance ofthe system, and to compare it to human experts. Each image was analysedindependently by the system and an experienced radiologist. The testingdataset was not seen by the system during the training phase.

The annotations made by the system and the experts were compared toevaluate the performance. A detailed analysis of all cases where humanand machine annotations were not in agreement was performed by anotherexperienced expert in dentomaxillofacial radiology to review possiblecauses of incorrect image interpretation. In such cases, the verifyingexpert had the final say to determine the ground truth. In the caseswhere the system and the expert provided the same annotations, both wereconsidered correct.

For the detection task, the human and machine annotations were deemed toagree if they intersected substantially. The remaining unmatched boxeswere composed of two error types: false positive results, whereredundant boxes were annotated, and false negative results, whereexistent teeth were missed. For the numbering task, human and machineannotations were deemed to agree if the class labels provided by expertsand the system for the same bounding boxes were identical.

Based on the results for detection and numbering tasks, metrics werecalculated to evaluate the performance of the system and the human. Forteeth detection, the following metrics were used:

${{precision} = \frac{TP}{{TP} + {FP}}}{{sensitivity} = \frac{TP}{{TP} + {FN}}}$

where TP, FP, and FN are true positive, false positive, and falsenegative, respectively. Accuracy in teeth numbering was calculated asthe ratio of correctly classified boxes to all boxes.

Teeth Detection

The above system implemented for teeth detection achieved a sensitivityof 0.9941 and a precision of 0.9945. The experts achieved a sensitivityof 0.9980 and a precision of 0.9998. The detailed data are presented inTable 1 below.

TABLE 1 System Expert True positives 5023 5043 False negatives 30 10False positives 28 1 Precision 0.9945 0.9998 Sensitivity 0.9941 0.9980

In general, the detection module 200 was found to demonstrate excellentresults, both for high-quality images with normal teeth arrangement andmore challenging cases such as overlapped or impacted teeth, images of apoor quality with blurred tooth contours, or teeth with crowns. It wasfound in the case of both teeth detection and numbering (classification)that errors made by the above-described system were due to similarfactors giving rise to errors by the experts. FIG. 2 shows successfuldetection examples for (a) all 32 teeth, (b) severely decayed andimpacted teeth, (c) exclusion of implants and detection of dentalcrowns, and (d) exclusion of cantilever elements of fixed bridges. Thedetected teeth are depicted with white boxes superimposed on the images.

In most cases, the detection module 200 correctly excluded bridges andimplants from the detection results. The main reasons for faultsincluded root remnants, presence of orthopaedic appliances, highlyimpacted and overlapped teeth. The system produced false positiveresults in the form of incorrectly detected implants and bridges, extraboxes for teeth with orthopaedic constructions and multiple-rooted teethand detected fragments outside of the jaw. Most human errors were falsenegatives caused by missed root remnants, probably as a result of lackof concentration.

Teeth Numbering

The teeth classification by the classification module 500 achieved asensitivity of 0.9800 and precision of 0.9994, while the expertsachieved a sensitivity of 0.9893 and a precision of 0.9997. The detaileddata are presented in Table 2 below. The Error statistics providedetails on three groups of misdiagnosed teeth: 1 tooth distance(neighboring teeth were misclassified), >1 (the predicted number wasmore than 1 tooth apart from the correct number), confused jaws (upperand lower jaws were confused).

TABLE 2 System Expert True positives 4938 4985 True negatives 156108156155 False negatives 101 54 False positives 101 54 Specificity 0.99940.9997 Sensitivity 0.9800 0.9893

FIG. 3 shows successful classification examples for (a) all 32 teeth,(b) severely decayed and impacted teeth, (c) teeth with dental crowns,and (d) correct classification despite missed teeth and lack of context.Again, the detected teeth are indicated by white rectangles. Thenumbering in these examples conforms to FDI notation.

It was found that extending the region of cropped teeth to includeadditional context and augmenting the images resulted in approximately a6 percentage point (pp) and a 2 pp increase of accuracy, respectively.The heuristic method based on spatial teeth number arrangement rulesincreased the accuracy by 0.5 pp.

The main reasons for numbering errors by the classification module 500includes lack of nearby teeth near the target tooth, too small remainingtooth fragments (root remnants or severely decayed teeth), and evidenceof extensive dental work. In most errors, the system confused a toothwith a missing adjacent one. Mainly, molars were misclassified. The samecases are reported by human experts to be challenging.

The foregoing results compare favourably to other studies, including LinP L, Lai Y H, Huang P W, “An effective classification and numberingsystem for dental bitewing radiographs using teeth region and contourinformation”, Pattern Recognit. 2010 43(4) pp. 1380-1392; Hosntalab M,Zoroofi R A, Tehrani-Fard A A, Shirani G, “Classification and numberingof teeth in multi-slice CT images using wavelet-Fourier descriptor”, IntJ Comput Assist Radiol Surg., 2010 5(3), p. 237-249; Mild Y, MuramatsuC, Hayashi T, Zhou X, Hara T, Katsumata A, et al., “Classification ofteeth in cone-beam CT using deep convolutional neural network”, ComputBiol Med. 2016 80, pp. 24-29.

Segmentation techniques may be implemented for more accuratelocalization. While the example above employed the CNN architecture,those skilled in the art will recognize the implementation of the abovesystem and methodology need not be limited to CNNs and that otherarchitectures and networks may be employed to further improve theaccuracy of this example, especially for diagnostics of pathologies. Itwill also be appreciated by those skilled in the art that one advantageof the CNN approach is that these improvement steps can be gradual, andresults of previous steps can be reused in the form of transferlearning: fine-tuning of existing models, training new models on alreadyannotated datasets, segmentation or localization of objects withinpreviously detected boundaries. It will also be appreciated that thesystem and methodology described herein can be employed to automatesteps in diagnosis.

Implementation Example

The foregoing deep learning system can be implemented in a networkedplatform to provide computer-aided diagnostics and dental charting usingradiographs or other dental images generated by one or more clientsystems to not only detect and number teeth, but also detect andidentify conditions of individual teeth. Services provided by thenetworked system can include automatic completion of dental charts;automatic identification of areas of interest in dental images forprofessionals to investigate and potentially diagnose; and sharing ofimages, data and reports between dental practitioners and other parties,subject to applicable privacy constraints. These services may save timeand reduce the negative effects associated with stress and fatigue indental practice on the accurate analysis of dental imaging.

FIG. 4 illustrates an example network environment for the deep learningsystem. In this example, the detection and classification modules may beimplemented in a cloud-based or networked analysis service 1000communicating with one or more client systems. The cloud-based service1000 is preferably operated in compliance with applicable privacylegislation, such as the United States Health Insurance Portability andAccountability Act of 1996 (HIPAA). Individual client systems 1070 maybe computer systems in dental offices communicating with the cloud-basedservice 1000 via a wide area network. The client system 1070communicates with the analysis service 1000 over a network to receiveupdated charting data and reports and to send responses (e.g., changesto provisional diagnoses), as described below. In one implementation, aweb server (not shown) provides an interface between the analysis system1000 and a browser client application executing on the client system1070. Data supplied by the analysis system 1000 may be presented ingraphical user interfaces as illustrated in FIGS. 5-9.

In a typical dental practice, it is expected that practitionersimplement practice management software for the creation and managementof patient data, including dental images. Individual dental practicesmay also establish their own price lists for services. This data(patient data, including dental images, and pricing data) may be storedlocally in a client system 1070, or remotely by a practice managementsystem 1052 and in image data storage 1054 and pricing data storage 1056in a dental support infrastructure 1050. The dental supportinfrastructure 1050 may comprise a remote server system or cloud-basedsystem, accessible by the client system 1070 via the Internet or othernetwork. The infrastructure 1050 may be operated or hosted by a thirdparty provider (e.g., a dental support organization); although in someimplementations, the dental support infrastructure 1050, or one or morecomponents of the infrastructure 1050 (e.g., the practice managementsystem 1052) may be hosted locally in a dental office, e.g., co-locatedwith the client system 1070. The infrastructure 1050 in turn may alsocommunicate with the analysis service 1000 to provide the pricing dataand image data to the analysis service 1000. The client system 1070receives results generated by the analysis service 1000 through a webserver of the analysis system 1000, as mentioned above, althoughalternatively the analysis service 1000 provides its results to theclient system 1070 via the practice management system 1052.

As another example implementation, one or more components of the dentalsupport infrastructure 1050 may be implemented in a cloud-based ornetworked system, integrated with the analysis system 1000. For example,the client system 1070 may access the dental support infrastructurecomponents via a special-purpose computer program or web browserexecuting on the client system 1070. The special-purpose computerprogram or web browser communicates with a web server (not shown) thatgenerates and transmits patient, analytical, and image data, includingdata generated by the analysis system 1000, to the client system 1070.Responses from the client system 1070 are sent to the dental supportinfrastructure 1050 and analysis system 1000 via the web server.

Communications between various components and elements of the networkenvironment (analysis system 1000, dental support infrastructure 1050and its components, the client system 1070) may occur over private orpublic connections, preferably with adequate security safeguards as areknown in the art. In particular, if communications take place over apublic network such as the Internet, suitable encryption is employed tosafeguard the privacy of data exchanged between the various componentsof the network environment.

The service 200 implements the detection and classification modules 200,500 described above, also preferably handling data in compliance withapplicable privacy legislation. These various services and systems 1000,1052, 1054, 1056, 1050 may be implemented by one or more servers,executing together in the same cloud-based environment; alternativelythe practice management and data storage systems may reside on serversoutside the cloud environment which communicate with the service 1000over the same wide area network as the cloud environment, or overanother network. The configuration of such servers and systems,including required processors and memory, operating systems, networkcommunication subsystems, and the like, will be known to those skilledin the art.

In the example of FIG. 4, client systems 1070 interact with a remotepractice management system 1052 for the management of patient data,pricing, and billing as well as viewing radiographic images. Images 1065are collected at the client site using radiographic or other imagingequipment. The images are transferred to the image data storage 1054over a network, for example using an application programming interface(API) 1060 to access the image data storage service. The images that aretransmitted may be accompanied by metadata. It is expected that imageswill be generated in digital format and therefore ready for transmissionover the network to the image data storage 1054; so, if radiographicimages are generated on film, then as mentioned above, the images willbe digitized first prior to transmission. The client system 1070 maysubsequently retrieve the images 1065 from the image data storage 1054for display locally. Optionally, pricing data may be provided by theclient system 1070 to the pricing data storage 1056. The pricing datamay then subsequently be transmitted (1) to the analysis service 1000for use in generating responsive data, as discussed below. In addition,the collected image data and associated metadata (e.g., sufficientinformation to associate the image with a patient; numbering information(e.g., if the image is one of a series of images); the date of theimage; the type of image (e.g., bitewing, PV, etc.); and informationabout the location of sensors or film, for example in the case of abitewing radiograph) is transmitted (2) to the analysis service 1000.Additionally, patient data and charting data is transmitted for thepatient (3). The charting data may comprise the patient's dental chart(in standard notation) from a previous visit or from the latestexaminations if service 1000 is utilised for a quality assurancepurpose.

The analysis service 1000 implements detection and numbering of presentand absent teeth as described above, using the patient, charting, andimage data received from the dental support infrastructure 1050. Using asimilar methodology, the analysis service 1000 can also facilitate thedetection and treatment of conditions: pathological conditions (e.g.,missing teeth, caries, apical periodontitis, dental cysts),non-pathological conditions (e.g., restorations, crowns, implants,bridges, endodontic treatments) and post-treatment conditions (e.g.,overhanging restorations, endodontic underfillings and overfillings).This automated detection of conditions may either replace or supplementdetection and analysis of dental images by dental and medicalpractitioners. In addition to detecting and numbering teeth from inputimages, the analysis service 1000 is further trained and identifies andclassifies regions of interest within detected teeth, to enable thegeneration of a symbolic dental chart with conditions provisionallyidentified or diagnosed for delivery through a computer interface to thepractitioner, e.g., via the practitioner's practice management software.For example, the analysis service 1000 may enable rapid analysis ofdental X-ray images, highlighting of teeth with radiological findings,providing supporting data, and present preliminary findings forconfirmation or rejection by practitioners to generate final assessmentreports. Detection of pathological conditions and radiological findingscan be performed using, again, an appropriately trained CNN or otherneural network architecture. Alternatively, conditions and findings maybe diagnosed by a practitioner and input manually via a user inputinterface at a client system. The results generated by the analysisservice 1000 are provided in the form of updated charting data (4) tothe practice management system 1052 (as mentioned above, this may be inaccordance with a preferred standard notation), optionally withpreliminary diagnostic findings correlated to tooth numbers that wereautomatically determined by the service 1000. The updated charting datamay also comprise an indication of the bounding box or region ofinterest for the input image as identified during either the toothdetection or condition detection by the analysis service 1000. Thisregion of interest information may comprise coordinates defining eachportion of the original image (e.g., coordinates identifying absolutepixel positions within the image, or coordinates and offsets defining arectangular region within the image) for which a tooth and/or acondition was detected, associated with a corresponding tooth number. Ifa condition was determined from the region of interest identified by thecoordinates, then the updated charting data also comprises an identifierof the condition associated with the tooth number and the coordinates.

In these examples, charting and patient data may be maintained in anelectronic form and handled in conformance with privacy requirements andestablished standards such as ANSI/ADA 1067:2013, Electronic DentalRecord System Standard Functional Requirements. Further, the datapreferably conforms to one or more established standards or conventionsincluding the aforementioned FDI and Universal numbering systems, andtransaction and code set standards as may be defined for dentalpractice, such as codes on dental nomenclature and coding mandated byHIPAA or other equivalent legislation or regulation, or otherstandardized codes such as the Code on Dental Procedures andNomenclature (CDT Code) published by the American Dental Association.

In one implementation, the updated charting data may be displayed in agraphical user interface (GUI) on a chair-side display of the clientsystem 1070 to practitioners and/or patients to visualise diagnosticfindings. This data and GUI may be served from the analysis system 1000directly to the client system 1070 (5). After practitioners confirm,rule out or add new findings based on clinical examinations and theirprofessional opinions, assessment reports with supporting data,including the ability to download an X-Ray or other image, andpractitioner recommendations may be generated by the analysis system1000 and transmitted (5) to the client system 1070 for sharing withpatients electronically. It will be appreciated from the followingexamples that automatic detecting and numbering the teeth fromradiographs and other types of input images by the analysis system 1000permits the generation of symbolic dental chart with associatedannotations correlating one or more specific conditions, as identifiedby the analysis system 1000 and the association of detected conditionsto a tooth number.

FIGS. 5 to 7 illustrate a number of example GUIs 1100 that may bedisplayed by a client program executing at the client system 1070. Apreliminary assessment generated by the service 1000 may be transmittedto the client system 1070 and rendered in a GUI such as GUI 1100. Inthis example, the service 1000 not only performs the detection andclassification (numbering) described above, but also detects andclassifies non-pathological and pathological conditions from the images.The classification (numbering) and detected conditions are identified inthe updated charting data transmitted to the practice management system1052 (4). This information can be retrieved, along with the image data,by the client system 1070 and displayed in a GUI such as the GUI 1100.The first example GUI of FIG. 5 comprises a patient information displayarea 1102 (in the figures, hypothetical patient data is shown); imagemanipulation interface 1104 (illustrated schematically), which mayinclude different user interface elements such as buttons, sliders, etc.for manipulating the dental image displayed in the image area 1106. Forexample, user interface elements may be provided for altering thebrightness and contrast of the image; zooming in or out; panning indifferent directions; marking up the image; and so on. The GUI may alsobe provided with a menu bar (not illustrated in the figures) forcarrying out various tasks such as saving altered images; sendingreports or images to a recipient; loading new patient information; andso on. Since a series of images may have been generated for a givenpatient at a single consultation, the GUI can include navigationelements 1108 for navigating to other images in the series.

The example GUI of FIG. 5 also presents the detected non-pathologicaland pathological conditions for the patient in different forms. Inaddition to the patient's PV image displayed in image area 1106, thedetected conditions can be indicated by bounding boxes or other visualelements on the displayed image. The locations and dimensions of thebounding boxes or visual elements may be determined by the regions ofinterest previously identified by the analysis service 1000, and can bevisually distinguished according to the type of condition detected. Forexample, non-pathological conditions may be marked on the image in afirst color, while pathological conditions are marked on the image usinga different color. In the accompanying figures, non-pathologicalconditions are indicated with a solid line rectangle, while pathologicalconditions are indicated in a dashed line rectangle. The GUI 1100 mayalso include user interface elements to permit the user to show or hidevisual elements of a certain type (e.g., to hide all thenon-pathological visual elements).

Furthermore, a standard symbolic numbered dental chart 1110 is alsodisplayed, comprising a plurality of teeth representing the typicalarrangement of a full complement of adult teeth (or primary teeth asappropriate) mapped or correlated to tooth position, number(classification), and detected conditions. The dental chart may also becolor coded to signal the location of non-pathological and pathologicalconditions. The color coding may match color coding used elsewhere inthe GUI (e.g., in the image region 1106). In addition, teeth that appearin the standard chart that are not detected (e.g., because the patientlacks those teeth) may be shown greyed-out. In some implementations ofthe GUI, a user may select (e.g., using a pointing device or touchinterface device, such as a touchscreen) a single tooth in the chart,and the detected conditions may be displayed adjacent to the tooth.Thus, at least some teeth depicted symbolically in the dental chart 1110are correlated to the regions of interest identified by the boundingboxes or other visual elements in the displayed image in image area1106.

In addition, a listing 1112 of the detected conditions is also included.in the example of FIG. 5, the list entries are correlated to acorresponding tooth by number and a confidence level for the detectedcondition (e.g. “13 Caries 99%” indicates that a cavity was detected intooth number 13 with a confidence level of 99%). Teeth that areidentified as present in the dental chart but otherwise exhibit nopathological or non-pathological conditions (i.e., “normal” teeth) maybe omitted from the listing of the detected conditions, or may beincluded in the listing and indicated as being normal. The mapping orcorrelation to the chart 1110 may be carried out either by the service1000 and transmitted as part of the GUI data to the client system 1070,or alternatively the mapping or correlation may be constructed by theclient system 1070 on receipt of the listing data.

The listing may include user interface elements, such as checkboxes, forthe practitioner to confirm or reject (delete) the findings made by theservice 1000. Confirmations and rejections, if submitted by thepractitioner, are sent to the service 1000 as feedback (again, this maybe in the form of charting data (3)) and may be used to provideadditional training to the CNN or other neural network.

FIG. 6 is a further example of the GUI 1100, displaying a bitewingradiograph in the image area 1106. In this case, as a bitewing imagecaptures only a subset of a patient's teeth, not every tooth in thestandard dental chart 1110 is visible in the image. Accordingly, thestandard dental chart 1110 may grey-out those teeth that are notcurrently visible. Additional navigation elements 1114 may also beprovided adjacent the standard dental chart 1110 to permit the user toswitch to an image displaying adjacent teeth, assuming that a series ofimages for the patient is available. Since the analysis system 1000 isable to determine the number of teeth in the series of images, the nextimage in the series can be identified and retrieved.

In the example of FIG. 6, the listing 1112 includes options for the userto filter 1116 the displayed results (e.g., to display caries only, orall detected pathological conditions, etc.). Filtering may also be usedto exclude detected conditions having an associated confidence levellower than a selected threshold. In this example, a slider 1117 isprovided to permit the user to input a desired confidence levelthreshold.

Since CNNs rely only on “raw” image data, and do not rely onhand-crafted features or special-purpose programming to be able todetect features in the dental images, the techniques described aboveextend to the interpretation of various other types of dental X-rayimages such as periapical intraoral radiographs, full-mouth series, or3D images such as cone beam computed tomographs, and may be extended tocraniofacial X-ray images such as cephalograms, as well as intraoral andextraoral camera images, videos, and 3D scans. FIGS. 7, 8, and 9illustrate further example GUIs 1100 containing the results of aperiapical image analysis, a bitewing image analysis, and a filteredbitewing image analysis, respectively. FIG. 7 includes a listing ofprobable pathological, non-pathological and problematic post-treatmentconditions; FIG. 8 lists probable pathological and non-pathologicalconditions, and the listing of FIG. 9 is filtered to show only probablepathological conditions. Again, the listings may include correspondingconfidence levels and provide an option for the practitioner to confirmor reject these findings.

As mentioned above, if the practitioner confirms or rejects findings,these responses may be transmitted to the analysis service 1000 forincorporation into training data for future analyses. In addition, theresults displayed in the GUI 1100 are updated to remove any rejectedentries. As shown in FIGS. 7 to 9, in place of the dental chart 1114 anoption to input new findings is provided. These added findings (toothnumber, diagnosis) can also be transmitted as new charting data (3) tothe analysis service for incorporation into training data. The listing112 is also correspondingly updated.

Once the practitioner has completed their review of the reportedfindings in the GUI, different forms of reports may be generated. As oneexample, an orthopantomogram report may be generated based on thediagnosis generated from a PV image. An example of a report 1120 isshown in FIG. 10. This report includes a finalized version of thestandard dental chart 1122; a listing, by tooth number (Americannotation is employed in FIG. 19) of teeth that were determined to bepresent (“Normal appearance”) and missing, together with a listing ofthe findings as confirmed by the practitioner 1124. In this report 1120,which is displayable by the client system 1070, fields are available forthe practitioner to enter comments. The report 1120 also includes imagescropped from the original PV image for each tooth having a correspondingfinding 1126. The report 1120 thus presents the detected conditions (asconfirmed or edited by the practitioner) in multiple formats.

Based on the detected pathological conditions the service 1000 canadditionally generate treatment planning data and transmit this data tothe practice management system 1052 for retrieval by the client system1070. Treatment planning data may include, at a minimum, a toothidentification (e.g., number) for each tooth having a detectedpathological condition and a prescribed treatment for the pathologicalcondition (the correlation between the prescribed treatment and thepathological condition may be included in the configuration records).The treatment planning data can be updated and transmitted again inresponse to receipt by the system 1000 of changes to the diagnosedconditions as a result of the practitioner rejecting a diagnosis, oradding a new finding.

Furthermore, if pricing data is provided (1) to the analysis service1000, based on the treatment planning data and patient data recorded bythe client systems concerning treatments actually carried out, theanalysis system 1000 can also track potential revenue from projectedtreatments (e.g., by aggregating the treatment planning data) and actualrevenue from treatments.

Another possible report is a patient education document or GUI view,depicting the projected costs of treatment for a given tooth dependingon the time of treatment. An example interface 1200 is depicted in FIG.9, for a scenario in which a caries is detected in a tooth. An initialprojected cost for treating the caries immediately with a filling may be$100 (as an example). However, if the patient chooses not to treat thecondition immediately, in 1-2 years, both a filling and onlay may berequired, with an estimated cost of $3000. In 2-3 years, a filling, rootcanal, and crown may be required, for a cost of $1600. If the conditionis left untreated for 3 years or more, extraction may be required,resulting in the need for an implant and crown, for a total estimatedcost of $3950. Based on the identification and detection of toothconditions by the analysis system 1000, the analysis system 1000 canidentify a corresponding treatment and retrieve the pricing data fromthe pricing data storage 1056 to compute a provisional cost oftreatment. Further, the analysis system 1000 can compute a projecteddiagnosis for the tooth if left untreated based on the tooth numberingand the condition of adjacent teeth, and retrieve costs associated withtreatment at specified time intervals to provide a projected cost oftreatment at later dates. This information can be transmitted to theclient system 1070 and displayed together with standardized graphicsrepresenting the process of the pathological condition over time toeffectively educate the patient on the options for treatment, and theconsequences of non-treatment, as depicted in 1200.

These systems and methods may further enable batch analysis of dentalimaging to, for example, a) compare the information in existing dentalcharts and dental records with the content of a practice's patients'dental images, and b) identify post-treatment conditions inpost-treatment images. For example, a practice or a dental plan providermay review all or a subset of its patient files to identify incompletedental charts, probable missed pathological findings or incorrectlydetected, as well as treatment plans that are not in accordance to thestandard of care, as part of its pre-treatment quality assuranceprocess. Also, a practice or a dental plan provider may review all or asubset of its patient files to identify problematic or exemplaryprocedures as part of its post-treatment quality assurance process. Theimages may be sorted by the probability of discrepancies or problematicpost-treatment findings and presented to a qualified professional forfurther analysis, as shown in the example GUIs or in the form of datasupplied to third-party analytics and business intelligence systems.

In still another aspect, these systems and methods may enable theprovision of objective assessments to support practitioner findings.This may engender greater patient trust in practitioner decision-making,and may educate and encourage patients to obtain treatment for diagnosedconditions.

The examples and embodiments are presented only by way of example andare not meant to limit the scope of the subject matter described herein.Variations of these examples and embodiments will be apparent to thosein the art and are considered to be within the scope of the subjectmatter described herein. For example, some steps or acts in a process ormethod may be reordered or omitted, and features and aspects describedin respect of one embodiment may be incorporated into other describedembodiments.

The data employed by the systems, devices, and methods described hereinmay be stored in one or more data stores. The data stores can be of manydifferent types of storage devices and programming constructs, such asRAM, ROM, flash memory, programming data structures, programmingvariables, and so forth. Code adapted to provide the systems and methodsdescribed above may be provided on many different types ofcomputer-readable media including computer storage mechanisms (e.g.,CD-ROM, diskette, RAM, flash memory, computer's hard drive, etc.) thatcontain instructions for use in execution by one or more processors toperform the operations described herein. The media on which the code maybe provided is generally considered to be non-transitory or physical.

Computer components, software modules, engines, functions, and datastructures may be connected directly or indirectly to each other inorder to allow the flow of data needed for their operations. Variousfunctional units have been expressly or implicitly described as modules,engines, or similar terminology, in order to more particularly emphasizetheir independent implementation and operation. Such units may beimplemented in a unit of code, a subroutine unit, object (as in anobject-oriented paradigm), applet, script or other form of code. Suchfunctional units may also be implemented in hardware circuits comprisingcustom VLSI circuits or gate arrays; field-programmable gate arrays;programmable array logic; programmable logic devices; commerciallyavailable logic chips, transistors, and other such components.Functional units need not be physically located together, but may residein different locations, such as over several electronic devices ormemory devices, capable of being logically joined for execution.Functional units may also be implemented as combinations of software andhardware, such as a processor operating on a set of operational data orinstructions.

It should also be understood that steps and the order of the steps inthe processes and methods described herein may be altered, modifiedand/or augmented and still achieve the desired outcome. Throughout thespecification, terms such as “may” and “can” are used interchangeably.Use of any particular term should not be construed as limiting the scopeor requiring experimentation to implement the claimed subject matter orembodiments described herein. Any suggestion of substitutability of thedata processing systems or environments for other implementation meansshould not be construed as an admission that the invention(s) describedherein are abstract, or that the data processing systems or theircomponents are non-essential to the invention(s) described herein.Further, while this disclosure may have articulated specific technicalproblems that are addressed by the invention(s), the disclosure is notintended to be limiting in this regard; the person of ordinary skill inthe art will readily recognize other technical problems addressed by theinvention(s).

1. A computer-implemented diagnostic method, comprising: obtaining adental image comprising one or more teeth; obtaining a tooth number foreach of the one or more teeth comprised in the image, wherein theobtaining comprises: detecting, using a computer system executing aneural network, the one or more teeth in the dental image; determiningfor each of one or more teeth, using the computer system, a tooth numberassociated with the tooth; and transmitting, to a client system, datacorrelating at least one region of the dental image with a correspondingtooth number.